MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, without focusing on token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and summarize our observations as follows: (1) MetaFormer ensures solid lower bound of performance. By merely adopting identity mapping as the token mixer, the MetaFormer model, termed IdentityFormer, achieves >80% accuracy on ImageNet-1K. (2) MetaFormer works well with arbitrary token mixers. When specifying the token mixer as even a random matrix to mix tokens, the resulting model RandFormer yields an accuracy of >81%, outperforming IdentityFormer. Rest assured of MetaFormer's results when new token mixers are adopted. (3) MetaFormer effortlessly offers state-of-the-art results. With just conventional token mixers dated back five years ago, the models instantiated from MetaFormer already beat state of the art. (a) ConvFormer outperforms ConvNeXt. Taking the common depthwise separable convolutions as the token mixer, the model termed ConvFormer, which can be regarded as pure CNNs, outperforms the strong CNN model ConvNeXt. (b) CAFormer sets new record on ImageNet-1K. By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1K: it achieves an accuracy of 85.5% at 224x224 resolution, under normal supervised training without external data or distillation. In our expedition to probe MetaFormer, we also find that a new activation, StarReLU, reduces 71% FLOPs of activation compared with GELU yet achieves better performance. We expect StarReLU to find great potential in MetaFormer-like models alongside other neural networks.

Pastic分割结合了语义和实例细分的优势，可以为智能车辆提供像素级和实例级别的环境感知信息。但是，它挑战各种尺度的对象，尤其是在极小的和小的物体上。在这项工作中，我们提出了两个轻量级模块来减轻此问题。首先，Pixel-ReSation Block旨在为大规模事物建模全局上下文信息，该信息基于与查询无关的公式，并带来小参数增量。然后，构建对流网络以收集针对小规模内容的额外高分辨率信息，为下游分割分支提供更合适的语义功能。基于这两个模块，我们提出了一个端到端尺度意识到的统一网络（Sunet），该网络更适合多尺度对象。对城市景观和可可的广泛实验证明了所提出的方法的有效性。

语义细分是智能车辆了解环境的重要任务。当前的深度学习方法需要大量的标记数据进行培训。手动注释很昂贵，而模拟器可以提供准确的注释。但是，在实际场景中应用时，使用模拟器数据训练的语义分割模型的性能将大大降低。对于语义分割的无监督域适应性（UDA）最近引起了越来越多的研究注意力，旨在减少域间隙并改善目标域的性能。在本文中，我们提出了一种新型的基于两阶段熵的UDA方法，用于语义分割。在第一阶段，我们设计了一个阈值适应的无监督局灶性损失，以使目标域中的预测正常，该预测具有轻度的梯度中和机制，并减轻了在基于熵方法中几乎没有优化硬样品的问题。在第二阶段，我们引入了一种名为跨域图像混合（CIM）的数据增强方法，以弥合两个域的语义知识。我们的方法在合成景观和gta5-to-cityscapes上使用DeepLabV2和使用轻量级的Bisenet实现了最新的58.4％和59.6％的MIOS和59.6％的Mious。

变压器在计算机视觉任务中表现出很大的潜力。常见的信念是他们的注意力令牌混合器模块对他们的能力做出了贡献。但是，最近的作品显示了变压器中的基于关注的模块可以被空间MLP所取代，由此产生的模型仍然表现得很好。基于该观察，我们假设变压器的一般架构，而不是特定的令牌混音器模块对模型的性能更为必要。为了验证这一点，我们刻意用尴尬的简单空间池汇集操作员取代变压器中的注意模块，以仅进行最基本的令牌混合。令人惊讶的是，我们观察到，派生模型称为池，在多台计算机视觉任务上实现了竞争性能。例如，在ImageNet-1K上，泳池制造器实现了82.1％的前1个精度，超越了调节的视觉变压器/ MLP样基线Deit-B / ResmmP-B24，比参数的35％/ 52％的准确度为0.3％/ 1.1％和48％/ 60％的Mac。泳道的有效性验证了我们的假设，并敦促我们启动“MetaFormer”的概念，这是一个从变压器抽象的一般架构，而无需指定令牌混音器。基于广泛的实验，我们认为MetaFormer是在视觉任务上实现最近变压器和MLP样模型的优越结果的关键球员。这项工作要求更具未来的研究，专门用于改善元形器，而不是专注于令牌混音器模块。此外，我们提出的池更换器可以作为未来的MetaFormer架构设计的起始基线。代码可在https://github.com/sail-sg/poolformer使用

探讨了语言建模流行的变形金刚，用于近期解决视觉任务，例如，用于图像分类的视觉变压器（VIT）。 VIT模型将每个图像分成具有固定长度的令牌序列，然后应用多个变压器层以模拟它们的全局关系以进行分类。然而，当从像想象中的中型数据集上从头开始训练时，VIT对CNNS达到较差的性能。我们发现它是因为：1）输入图像的简单标记未能模拟相邻像素之间的重要局部结构，例如边缘和线路，导致训练采样效率低。 2）冗余注意骨干骨干设计对固定计算预算和有限的训练样本有限的具有限制性。为了克服这些限制，我们提出了一种新的令牌到令牌视觉变压器（T2T-VIT），它包含1）层 - 明智的代币（T2T）转换，通过递归聚合相邻来逐步地结构于令牌到令牌。代币进入一个令牌（令牌到令牌），这样可以建模由周围令牌所代表的本地结构，并且可以减少令牌长度; 2）一种高效的骨干，具有深度狭窄的结构，用于在实证研究后CNN建筑设计的激励变压器结构。值得注意的是，T2T-VIT将Vanilla Vit的参数计数和Mac减少了一半，同时从想象中从头开始训练时，改善了超过3.0 \％。它还优于Endnets并通过直接培训Imagenet训练来实现与MobileNets相当的性能。例如，T2T-VTO与Reset50（21.5M参数）的可比大小（21.5M参数）可以在图像分辨率384 $ \ Times 384上实现83.3 \％TOP1精度。 （代码：https://github.com/yitu-opensource/t2t-vit）

Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in the ResNet architecture with STM blcoks to form a simple yet effective STM network by introducing very limited extra computation cost. Extensive experiments demonstrate that the proposed STM network outperforms the state-of-the-art methods on both temporal-related datasets (i.e., Something-Something v1 & v2 and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51) with the help of encoding spatiotemporal and motion features together. * The work was done during an internship at SenseTime.

In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed implicitly, by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. CMT obtains 73.0% NDS on nuScenes benchmark. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code will be released at https://github.com/junjie18/CMT.

The development of social media user stance detection and bot detection methods rely heavily on large-scale and high-quality benchmarks. However, in addition to low annotation quality, existing benchmarks generally have incomplete user relationships, suppressing graph-based account detection research. To address these issues, we propose a Multi-Relational Graph-Based Twitter Account Detection Benchmark (MGTAB), the first standardized graph-based benchmark for account detection. To our knowledge, MGTAB was built based on the largest original data in the field, with over 1.55 million users and 130 million tweets. MGTAB contains 10,199 expert-annotated users and 7 types of relationships, ensuring high-quality annotation and diversified relations. In MGTAB, we extracted the 20 user property features with the greatest information gain and user tweet features as the user features. In addition, we performed a thorough evaluation of MGTAB and other public datasets. Our experiments found that graph-based approaches are generally more effective than feature-based approaches and perform better when introducing multiple relations. By analyzing experiment results, we identify effective approaches for account detection and provide potential future research directions in this field. Our benchmark and standardized evaluation procedures are freely available at: https://github.com/GraphDetec/MGTAB.

Learning feature interactions is the key to success for the large-scale CTR prediction and recommendation. In practice, handcrafted feature engineering usually requires exhaustive searching. In order to reduce the high cost of human efforts in feature engineering, researchers propose several deep neural networks (DNN)-based approaches to learn the feature interactions in an end-to-end fashion. However, existing methods either do not learn both vector-wise interactions and bit-wise interactions simultaneously, or fail to combine them in a controllable manner. In this paper, we propose a new model, xDeepInt, based on a novel network architecture called polynomial interaction network (PIN) which learns higher-order vector-wise interactions recursively. By integrating subspace-crossing mechanism, we enable xDeepInt to balance the mixture of vector-wise and bit-wise feature interactions at a bounded order. Based on the network architecture, we customize a combined optimization strategy to conduct feature selection and interaction selection. We implement the proposed model and evaluate the model performance on three real-world datasets. Our experiment results demonstrate the efficacy and effectiveness of xDeepInt over state-of-the-art models. We open-source the TensorFlow implementation of xDeepInt: https://github.com/yanyachen/xDeepInt.

In this paper, we study the problem of knowledge-intensive text-to-SQL, in which domain knowledge is necessary to parse expert questions into SQL queries over domain-specific tables. We formalize this scenario by building a new Chinese benchmark KnowSQL consisting of domain-specific questions covering various domains. We then address this problem by presenting formulaic knowledge, rather than by annotating additional data examples. More concretely, we construct a formulaic knowledge bank as a domain knowledge base and propose a framework (ReGrouP) to leverage this formulaic knowledge during parsing. Experiments using ReGrouP demonstrate a significant 28.2% improvement overall on KnowSQL.